Evaluating the Project54 Speech User Interface

The Project54 system integrates multiple electronic devices in police cruisers into a single system with a speech user interface (SUI). The system has been deployed in over 240 cruisers in the state of New Hampshire. We collected over 49,000 samples of speech commands, along with the corresponding SUI responses, from 27 officers using the Project54 system during their everyday work. We found that the system recognized officer utterances around 85% of the time. About one third of recognition errors were due to speech recognizer errors and about two thirds were due to user errors. Three types of user errors were identified: issuing a command that is not valid in any context of the SUI (54% of user errors), issuing a command that is valid in some context but not in the current one (34%) and operating the push-to-talk button incorrectly (12%).